<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.11"/>
<title>Superpixel Benchmark: Executables</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/javascript">
  $(document).ready(function() { init_search(); });
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectalign" style="padding-left: 0.5em;">
   <div id="projectname">Superpixel Benchmark
   </div>
   <div id="projectbrief">Superpixel benchmark, tools and algorithms.</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.11 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search",false,'Search');
</script>
  <div id="navrow1" class="tabs">
    <ul class="tablist">
      <li><a href="index.html"><span>Main&#160;Page</span></a></li>
      <li class="current"><a href="pages.html"><span>Related&#160;Pages</span></a></li>
      <li><a href="annotated.html"><span>Classes</span></a></li>
      <li><a href="files.html"><span>Files</span></a></li>
      <li>
        <div id="MSearchBox" class="MSearchBoxInactive">
        <span class="left">
          <img id="MSearchSelect" src="search/mag_sel.png"
               onmouseover="return searchBox.OnSearchSelectShow()"
               onmouseout="return searchBox.OnSearchSelectHide()"
               alt=""/>
          <input type="text" id="MSearchField" value="Search" accesskey="S"
               onfocus="searchBox.OnSearchFieldFocus(true)" 
               onblur="searchBox.OnSearchFieldFocus(false)" 
               onkeyup="searchBox.OnSearchFieldChange(event)"/>
          </span><span class="right">
            <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
          </span>
        </div>
      </li>
    </ul>
  </div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Executables </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>As discussed in <a class="el" href="BUILDING_8md.html">Building</a> and depending on the chosen options, several executables are provided (usually found in <code>bin</code>). Most of these provide easy-to-use command line tools for the various algorithms. However, some utility command line tools for evaluation and visualization are also provided.</p>
<p>For algorithms written in MatLab, Bash scripts are provided to easily run the algorithms from the command line. This was mainly used for parameter optimization and evaluation. However, it is recommended to use the corresponding MatLab functions instead.</p>
<p>The same holds for PF, which is writen in Java, and WP, written in Python.</p>
<p>Table of contents:</p>
<ul>
<li><a href="#algorithms-in-c++">Algorithms in C++</a></li>
<li><a href="#utilities-in-c++">Utilities in C++</a><ul>
<li><a href="#eval_boundaries2labels_cli"><code>eval_boundaries2labels_cli</code></a></li>
<li><a href="#eval_connected_relabel_cli"><code>eval_connected_relabel_cli</code></a></li>
<li><a href="#eval_parameter_optimization"><code>eval_parameter_optimization</code></a></li>
<li><a href="#eval_summary_cli"><code>eval_summary_cli</code></a></li>
<li><a href="#eval_average_cli"><code>eval_average_cli</code></a></li>
<li><a href="#eval_visualization_cli"><code>eval_visualization_cli</code></a></li>
</ul>
</li>
<li><a href="#algorithms-in-matlab">Algorithms in MatLab</a></li>
<li><a href="#algorithms-in-java">Algorithms in Java</a></li>
<li><a href="#algorithms-in-python">Algorithms in Python</a></li>
</ul>
<h2>Algorithms in C++</h2>
<p>All command line tools for algorithms in C++ have the following options in common: </p><pre class="fragment">$ ../bin/cis_cli --help
Allowed options:
  -h [ --help ]                   produce help message
  -i [ --input ] arg              folder containing the images to process
  # Algorithm specific options ...
  -o [ --csv ] arg                save segmentation as CSV file
  -v [ --vis ] arg                visualize contours
  -x [ --prefix ] arg             output file prefix
  -w [ --wordy ]                  verbose/wordy/debug
</pre><p><code>--input</code> is additionally a positional option. The algorithm specific options can be displayed using the <code>--help</code> options. For details on the specific options, the reader is referred to the corresponding publication(s), see <a class="el" href="ALGORITHMS_8md.html">here</a> for references.</p>
<p>The <code>--csv</code> option will output the superpixel segmentations as <code>.csv</code> files in the provided directory, which is created if it does not exist. The naming follows the naming of the images found in the directory specified by <code>--input</code>. Similarly, <code>--vis</code> outputs visualizations in the provided directory, which is also created if it does not exist.</p>
<p><code>--prefix</code> can be used to specify a prefix, then the output files (CSV files and visualizations) are prefixed with the given string. <code>--wordy</code> will cause the tool to provide more detailed output while running (i.e. be verbose).</p>
<p>Examples: </p><pre class="fragment">$ build
$ cmake ..
$ make
# reSEEDS (built by default)
../bin/reseeds_cli --input ../data/BSDS500/images/test/ --superpixels 1200 --bins 5 --neighborhood 0 --confidence 0.1 --spatial-weight 0 --iterations 25 --color-space 0 -o ../output/reseeds -w
# ERGC (built by default)
$ ../bin/ergc_cli --input ../data/BSDS500/images/test/ --superpixels 1200 --color-space 1 --perturb-seeds 0 --compacity 0 -o ../output/ergc -w
# ETPS (built by default)
../bin/etps_cli --input ../data/BSDS500/images/test/ --superpixels 1200 --regularization-weight 0.01 --length-weight 0.1 --size-weight 1 --iterations 25 -o ../output/etps -w
</pre><h2>Utilities in C++</h2>
<p>As part of the benchmark, several tools for evaluation are provided. All of them are prefixed by <code>eval_</code> and only depend on <code>lib_eval</code>.</p>
<h3><code>eval_boundaries2labels_cli</code></h3>
<p><code>eval_boundaries2labels_cli</code> converts boundary maps saved as <code>.csv</code> files to superpixel segmentations as <code>.csv</code>: </p><pre class="fragment">$ ../bin/eval_boundaries2labels_cli --help
Allowed options:
  --help                     produce help message
  -i [ --input-csv ] arg     folder containing the CSV files to process
  -m [ --input-images ] arg  folder containing the corresponding images to 
                             process
  -r [ --overwrite ]         Overwrite original files
  -o [ --csv ] arg (=output) save segmentation as CSV file
  -w [ --wordy ]             wordy/verbose
</pre><p>Usage examples:</p>
<ul>
<li><code>examples/bash/run_tp.sh</code></li>
</ul>
<h3><code>eval_connected_relabel_cli</code></h3>
<p><code>eval_connected_relabel_cli</code> takes superpixel segmentations as <code>.csv</code> files and relabels them such that superpixels represent connected components. The original files can either be overwritten, or the relabeled superpixel segmentations can be saved in a separate directory: </p><pre class="fragment">$ ../bin/eval_connected_relabel_cli --help
Allowed options:
  --help                     produce help message
  -i [ --input-csv ] arg     folder containing the corresponding images to 
                             process
  -m [ --input-images ] arg  dummy option!
  -r [ --overwrite ]         Overwrite original files
  -o [ --csv ] arg (=output) save segmentation as CSV file
  -w [ --wordy ]             wordy/verbose
</pre><p>Usage examples:</p>
<ul>
<li><code>examples/bash/run_tp.sh</code></li>
</ul>
<h3><code>eval_parameter_optimization</code></h3>
<p><code>eval_parameter_optimization</code> demonstrates the parameter optimization procedure used in [1]. The parameters are partly the original parameters used in the paper. </p><pre class="fragment">[1] D. Stutz, A. Hermans, B. Leibe.
    Superpixels: An Evaluation of the State-of-the-Art.
    Computing Research Repository, abs/1612.01601.
</pre><p>Note that it should not be required to use this tool in most cases, unless parameters need to be optimized on new datasets or for new algorithms.</p>
<p>For parameter optimization, the command line tools for all algorithms are normed in the sense that they provide the parameters <code>-i</code> and <code>-o</code> for input and output. Having a close look at <code><a class="el" href="eval__parameter__optimization__cli_2main_8cpp.html">eval_parameter_optimization_cli/main.cpp</a></code> shows that the parameters for the different algorithms are hard-coded, e.g. </p><pre class="fragment">/**
 * Connector for parameter optimization of reSEEDS.
 * 
 * @param img_directory
 * @param gt_directory
 * @param base_directory
 */
void connector_reSEEDS(boost::filesystem::path img_directory, 
        boost::filesystem::path gt_directory, boost::filesystem::path base_directory,
        std::vector&lt;int&gt; superpixels) {

    for (unsigned int k = 0; k &lt; superpixels.size(); k++) {
        ParameterOptimizationTool tool(img_directory, gt_directory,
                base_directory / boost::filesystem::path(std::to_string(superpixels[k])),
            RELATIVE_PATH + "/bin/reseeds_cli", FAIR);

        tool.addIntegerParameter("superpixels", "--superpixels", std::vector&lt;int&gt;{superpixels[k]});
        tool.addIntegerParameter("bins", "--bins", std::vector&lt;int&gt;{1, 3, 5, 7}); // 4
        tool.addIntegerParameter("neighborhood", "--neighborhood", std::vector&lt;int&gt;{0, 1}); // 2
        tool.addFloatParameter("confidence", "--confidence", std::vector&lt;float&gt;{0.1f}); // 1
        tool.addFloatParameter("spatial-weight", "--spatial-weight", std::vector&lt;float&gt;{0.0f, 0.25f, 0.5f}); // 3
        tool.addIntegerParameter("iterations", "--iterations", std::vector&lt;int&gt;{1, 10, 25}); // 3
        tool.addIntegerParameter("color-space", "--color-space", std::vector&lt;int&gt;{0, 1, 2}); // 3

        tool.optimize();
    }
}
</pre><p>Also note that the Java path as well as the MatLab path and the directory containing the executables are hard-coded: </p><pre class="fragment">std::string RELATIVE_PATH = ".";
std::string MATLAB_EXECUTABLE = "/home/david/MATLAB/R2014b/bin/matlab";
std::string JAVA_EXECUTABLE = "/home/david/jdk-1.8.0_45/release/java";
</pre><p>The executable directory is <code>RELATIVE_PATH + "/bin"</code>. Then, only the algorithm needs to be specified together with the remaining options: </p><pre class="fragment">$ ../bin/eval_parameter_optimization_cli --help
Allowed options:
  --img-directory arg                   image directory
  --gt-directory arg                    ground truth directory
  --base-directory arg                  base directory
  --algorithm arg                       algorithm to optimize: reseeds, 
  --depth-directory arg                 depth directory
  --intrinsics-directory arg            intrinsics directory
  --matlab-executable arg (=../../MATLAB/R2014b/release/matlab)
                                        matlab executable path
  --java-executable arg (=../../jdk-1.8.0_45/release/java)
                                        java executable
  --not-fair                            do not use fair parameters
  --help                                produce help message
</pre><h3><code>eval_summary_cli</code></h3>
<p><code>eval_summary_cli</code> may the most important tool provided. It bundles all evaluation metrics. Given a directory containing superpixel segmentations as <code>.csv</code> files and directories with the corresponding images (as <code>.png</code>, <code>.jpg</code> or <code>:jpeg</code>) and ground truth segmentations (also as <code>.csv</code> files), summarizes the performance of the superpixel segmentations. The provided options are: </p><pre class="fragment">$ ../bin/eval_summary_cli --help
Allowed options:
  --sp-directory arg    superpixel segmentation directory
  --img-directory arg   image directory
  --gt-directory arg    ground truth directory
  --append-file arg     append file
  --vis                 visualize results
  --help                produce help message
</pre><p>Usage examples can be found in <code>examples/bash</code>. For <code>examples/bash/run_reseeds.sh</code> the created summary looks as follows: </p><pre class="fragment">metric,mean[0],mean[1],mean[2],mean[3],mean[4],mean_min,mean_max
ue,0.0289073,0.0303683,0.0298518,0.0306024,0.0354846,0.019728,0.0461874
oe,0.970836,0.971261,0.969193,0.967363,0.963669,0.954067,0.979919
rec,0.981173,0.976853,0.979881,0.981565,0.97807,0.965346,0.98946
pre,0.0580179,0.0563268,0.0588434,0.0611075,0.0646213,0.0412821,0.0831838
ue_np,0.0574405,0.0603835,0.0592958,0.0607787,0.0704587,0.0393508,0.0914618
ue_levin,9.14557,11.1276,13.0967,12.9077,12.8244,2.14357,28.7954
asa,0.971093,0.969632,0.970149,0.969398,0.964516,0.953813,0.980272
sse_rgb,583.115,583.115,583.115,583.115,583.669,578.606,578.606
sse_xy,84.9599,84.9599,84.9599,84.9599,85.0727,85.0815,85.0815
co,0.101888,0.101888,0.101888,0.101888,0.101926,0.101995,0.101995
ev,0.924642,0.924642,0.924642,0.924642,0.924634,0.925177,0.925177
mde,0.206509,0.232733,0.216243,0.20705,0.22207,0.163584,0.284383
icv,20.1905,20.1905,20.1905,20.1905,20.1899,20.1428,20.1428
cd,0.653127,0.653127,0.653127,0.653127,0.653058,0.65289,0.65289
reg,0.24697,0.24697,0.24697,0.24697,0.247334,0.247376,0.247376
sp,1201.33,1201.33,1201.33,1201.33,1201.33,1201.33,1201.33
sp_size,128.525,128.525,128.525,128.525,128.525,128.525,128.525
sp_size_variation,76.7064,76.7064,76.7064,76.7064,76.7555,76.7697,76.7697
</pre><p>Note that some statistics have been omitted here. The results can more easily be analyzed using Excel or Calc:</p>
<div class="image">
<img src="SUMMARY.png?raw=true"  alt="Evaluation summary." title="Evaluation summary."/>
</div>
<p>More usage examples:</p>
<ul>
<li><code>examples/bash/run_ccs.sh</code></li>
<li><code>examples/bash/run_cis.sh</code></li>
<li><code>examples/bash/run_crs.sh</code></li>
<li>...</li>
</ul>
<h3><code>eval_average_cli</code></h3>
<p><code>eval_average_cli</code> is used to calculate the average metrics given an evaluation summary. In the ideal case, the <code>--append-file</code> option for <code>eval_summary_cli</code> is used to gather the results for different numbers of superpixels in a single CSV file. The tool then offers the following options: </p><pre class="fragment">$ ../bin/eval_average_cli --help
Allowed options:
  --summary-file arg                    CSV summary file
  -o [ --output-file ] arg (=average.csv)
                                        output file
  --help  
</pre><p>The output might look as follows: </p><pre class="fragment">    K         Rec        1 - UE     EV
    188.291   0.602725   0.772888   0.717307
    296.472   0.658145   0.817905   0.754624
    387.935   0.692999   0.837564   0.773384
    608.065   0.749924   0.866199   0.802638
    794.477   0.783432   0.879694   0.816675
    1100.89   0.822392    0.89422   0.833498
    1302.86   0.843054   0.900908   0.841242
    1302.86   0.843054   0.900908   0.841242
    1572.26   0.865465   0.907163   0.849824
    1960.33   0.890398   0.913937   0.858543
    1960.33   0.890398   0.913937   0.858543
    2478.44   0.917414   0.921283   0.867261
    3292.52   0.945248   0.928409   0.876992
    3292.52   0.945248   0.928409   0.876992
    3292.52   0.945248   0.928409   0.876992
    4439.47   0.969279   0.934941   0.885738
    4439.47   0.969279   0.934941   0.885738
    6526.86    0.99025   0.943163   0.896176
 ---------- ---------- ---------- ----------
               10.4497    8.74869    14.1893
</pre><p>Where the final numbers correspond to the average metrics.</p>
<p>Usage examples:</p>
<ul>
<li><code>examples/bash/evaluate_w.sh</code></li>
<li><code>examples/bash/compare_fh_refh.sh</code></li>
<li><code>examples/bash/compare_seeds_reseeds.sh</code></li>
</ul>
<h3><code>eval_visualization_cli</code></h3>
<p><code>eval_visualization_cli</code> can be used to visualize superpixel segmentations available as <code>.csv</code> files given the corresponding images: </p><pre class="fragment">$ ../bin/eval_visualization_cli --help
Allowed options:
  --help                     produce help message
  --csv arg                  superpixel segmentation (as CSV)
  --images arg               image
  --contours                 draw contours
  --contours-on-white        draw contours on white image
  --means                    draw means
  --perturbed-means          draw perturbed means
  --random                   randomly color
  -v [ --vis ] arg (=output) output folder
  -x [ --prefix ] arg        input and output file prefix
  -w [ --wordy ]             verbose/wordy/debug
</pre><p>The offered visualizations are: drawing contours along superpixel boundaries, coloring the superpixels by their mean color, coloring the superpixels by their mean colors plus random noise, drawing the contours along superpixel boundaries on a white background, coloring the superpixels with random colors.</p>
<p>Example:</p>
<div class="image">
<img src="VISUALIZATIONS.png?raw=true"  alt="Visualizations." title="Visualizations."/>
</div>
<p>Usage examples:</p>
<ul>
<li><code>examples/bash/run_ccs.sh</code></li>
<li><code>examples/bash/run_cis.sh</code></li>
<li><code>examples/bash/run_crs.sh</code></li>
<li>...</li>
</ul>
<h2>Algorithms in MatLab</h2>
<p>For parameter optimization purposes, all algorithms implemented in MatLab provide both a Bash script and a MatLab function. For example, <code>tp_cli</code> contains both <code>tp_cli/tp_dispatcher.sh</code> and <code>tp_cli/tp_cli.m</code>.</p>
<p>The MatLab function usually provides the following parameters:</p>
<ul>
<li><code>folder</code>: the input folder containing the images to process.</li>
<li><code>csv_folder</code>: the output folder to save the superpixel segmentations as <code>.csv</code>.</li>
<li><code>vis_folder</code>: the output folder for visualizations.</li>
<li><code>prefix</code>: prefix used to output files (both CSV and visualizations).</li>
<li><code>wordy</code>: whether to run in verbose mode.</li>
</ul>
<p>Details on the algorithm specific algorithms can be found in the corresponding MatLab functions. Usually, all parameters after <code>folder</code> are optional.</p>
<p>Alternatively, the Bash file usually provides options similar to the options provided by the C++ command line tools:</p>
<ul>
<li><code>-i</code> is the input directory;</li>
<li><code>-o</code> the output CSV directory;</li>
<li><code>-v</code> the output visualization directory;</li>
<li>and <code>-h</code> for a help message.</li>
</ul>
<p>An examples: </p><pre class="fragment">$ ../tp_cli/tp_dispatcher.sh -h
Allowed options:
-i folder to process
# Algorithm specific options ...
-o output folder
-v folder for visualizatioN
-x prefix of output files
-w verbose output
-e path to MatLab executable
-a path to add, usually the lib_seaw path
</pre><p>Additionally, it is required to provide the path to the MatLab functions via <code>-a</code>, in this case the path to <code>tp_cli.m</code>, as well as the path to the MatLab executable via <code>-e</code>.</p>
<p>Examples of using the Bash scripts: </p><pre class="fragment">$ cd examples
# EAMS
$ ../eams_cli/eams_dispatcher.sh -i ../data/BSDS500/images/test/ -b 2 -m 50 -r 1 -o ../output/eams -w -a ../eams_cli -e ~/MATLAB/R2015b/bin/matlab
# POISE
$ ../poise_cli/poise_dispatcher.sh -i ../data/BSDS500/images/test/ -s 1200 -t 0.01 -g 2.5 -m 2 -c 0.6 -o ../output/poise -w -a ../poise_cli -e ~/MATLAB/R2015b/bin/matlab
</pre><p>Examples of using the MatLab functions directly: </p><pre class="fragment">% Options correspnd to the above call of EAMS.
folder = '../data/BSDS500/images/test/';
bandwidth = 2;
minimum_size = 50;
color_space = 1;
csv_folder = '../output/eams';
vis_folder = ''; # No visualization ...

fprintf('Running EAMS:\n');
fprintf('folder: %s\n', folder);
fprintf('bandwidth: %f\n', bandwidth);
fprintf('minimum_size: %d\n', minimum_size);
fprintf('color_space: %d\n', color_space);
fprintf('csv_folder: %s\n', csv_folder);
fprintf('vis_folder: %s\n', vis_folder);

try
    eams_cli(folder, bandwidth, minimum_size, color_space, csv_folder, vis_folder);
catch e
    fprintf([e.message '\n'])
end
</pre><p>More usage examples:</p>
<ul>
<li><code>examples/bash/run_eams.sh</code></li>
<li><code>examples/bash/run_nc.sh</code></li>
<li><code>examples/bash/run_poise.sh</code></li>
<li>...</li>
</ul>
<h2>Algorithms in Java</h2>
<p>PF is the only algorithm written in Java. A Bash script as described above is provided. Alternatively, the algorithm can be run as follows: </p><pre class="fragment">java -jar lib_pf/PathFinder.jar data/BSDS500/images/test output/pf 30 1200
</pre><h2>Algorithms in Python</h2>
<p>WP is the only algorithm written in Python. A Bash script as described above is provided. Alternatively, the algorithm can be called directly using Python: </p><pre class="fragment">python lib_wp/demo_waterpixels_smil_with_parser.py --original_image data/BSDS500/images/test/3096.jpg --superpixels 1200 --weight 10 --output output/wp/ </pre> </div></div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Sun Apr 16 2017 16:39:17 for Superpixel Benchmark by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/>
</a> 1.8.11
</small></address>
</body>
</html>
